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SELF ADJUSTING STEREO CAMERA SYSTEM 

BACKGROUND OF THE INVENTION 

1, Field of the Invention 

The present invention relates generally to 
5 stereo camera systems and, more particularly, to stereo 
camera systems which automatically self adjust to 
increase the resolution of an object of interest based on 
a detected distance of the object of interest from the 
camera and/or a detected size of the object of interest. 

10 2. Prior Art 

f* Various imaging systems have been developed in 

K 4 the art to produce stereoscopic or 3D images. These 

systems generally consist of a method for creating two 

^ images as seen from a different perspective and means for 

hh 15 displaying the images so that one eye sees one 

pbj perspective of the image and the other eye sees a 

H ; different perspective of the image to produce a 3D image. 

One method is based on the use of separate 
spaced apart optical means to produce two perspectives of 

20 the same image. The images are displayed side by side, 
such as on film or on a monitor. Means are provided so 
that the left eye sees only one image and the right eye 
sees only the other image so that the image as processed 
by the brain appears as a stereoscopic image. Thus, 

25 Stereographic photography is the method of producing 
images which are apparently three dimensional by 
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recording separate left- and right-eye images. The viewer 
reconstructs the 3-D image by viewing the two separate 2- 
D images simultaneously. 

Such stereo views have historically been 
5 created with a single camera and mirrors or with two or 
more cameras mounted on a platform. In such systems, 
parameters which affect the spatial resolution of the 
object of interest are generally adjustable, such as the 
pan and tilt of the cameras and the distance between the 
10 cameras (also known as the baseline) . However, in the 
Q prior art stereo camera systems, these parameters are 

adjusted before use and remain the same throughout the 
M period of operation, thus, no improvement m spatial 

[.J resolution is made during operation. The reason for 

15 selecting and fixing these parameters is a tradeoff 
s between spatial resolution and operational range, so as 

r*i to increase the space in which an object of interest can 

^ move about . 

In view of the prior art, there is a need for a 
20 stereo camera system, which resolves these and other 
problems with the prior art stereo camera systems. 

SUMMARY OF THE INVENTION 

Therefore it is an object of the present 
invention to provide a stereo camera system which 
25 improves spatial resolution during the mode of operation 
of the stereo camera system but which does not decrease 
the operational range of the system. 
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Accordingly, A stereo camera system is 
provided. The stereo camera system comprises: a stereo 
imaging means for outputting at least one stereo image; 
recognition means for locating an object of interest in 
the field of view of the stereo imaging means and at 
least one of a distance of the object of interest from 
the stereo imaging means and the size of the object of 
interest; and adjusting means for automatically changing 
at least one system parameter which affects the spatial 
resolution of the object of interest based on at least 
one of the located distance of the object of interest 
from the stereo imaging means and the size of the object 
of interest. 

In a first configuration of the stereo camera 
system, the stereo imaging means comprises: a camera; and 
a set of mirrors angled with respect to each other at a 
predetermined angle and disposed a predetermined distance 
from the camera for producing a stereo effect in the 
output of the camera. In which case, the adjusting means 
preferably comprises at least one of: angle adjustment 
means for adjusting the predetermined angle between the 
set of mirrors; distance adjustment means for adjusting 
the predetermined distance between the camera and the set 
of mirrors; and focal length adjustment means for 
changing a focal length of the camera. 

In a second configuration of the stereo camera 
system, the stereo imaging means comprises two or more 
cameras, each camera being angled a predetermined angle 
and distanced a predetermined distance with respect to 
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each other and the object of interest. In which case, 
the adjusting means preferably comprises at least one of: 
angle adjustment means for adjusting the predetermined 
angle of at least one of the two or more cameras; 
baseline adjustment means for adjusting the predetermined 
distance between the two or more cameras; distance 
adjusting means for adjusting a distance between at least 
one of the two or more cameras and the object of 
interest; and focal length adjustment means for changing 
a focal length of at least one of the two or more 
cameras . 

In either of the first or second configurations 
of the stereo camera system, the cameras can be still 
cameras where the at least one stereo image is a still 
image or video cameras where the at least one stereo 
image is a sequence of video images. 

Additionally, in either of the first or second 
configurations of the stereo camera system, the same 
preferably further comprises a controller for controlling 
at least one of the angle, distance, and focal length 
adjustment means based on an input signal from the 
recognition means. The recognition means is preferably a 
stereo vision system. 

Also provided is a stereo camera system for use 
with a stereo imaging means, such as the first and second 
configurations discussed above. The stereo camera system 
comprising: recognition means for locating an object of 
interest in the field of view of the stereo imaging means 
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and at least one of a distance of the object of interest 
from the stereo imaging means and the size of the object 
of interest; and adjusting means for automatically 
changing at least one system parameter which affects the 
spatial resolution of the object of interest based on at 
least one of the located distance of the object of 
interest from the stereo imaging means and the size of 
the object of interest. 

Still yet provided is a method for adjusting a 
stereo camera system to control spatial resolution of an 
object of interest in the field of view of a stereo 
imaging means. The method comprises the steps of: 
outputting at least one image from the stereo imaging 
means; locating an object of interest in the field of 
view of the stereo imaging means and at least one of the 
distance of the object of interest from the stereo 
imaging means and the size of the object of interest; and 
automatically changing at least one system parameter 
which affects the spatial resolution of the object of 
interest based on at least one of the located distance of 
the object of interest from the stereo imaging means and 
the size of the object of interest. 

BRIEF DESCRIPTION OF THE DRAWINGS 

These and other features, aspects, and 
advantages of the apparatus and methods of the present 
invention will become better understood with regard to 
the following description, appended claims, and 
accompanying drawings where: 
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Figure 1 illustrates a schematic representation 
of the stereo camera system of the present invention 

Figure 2 illustrates a schematic representation 
of the stereo camera system of Figure 1 having a first 
5 configuration of a stereo imaging means. 

Figure 3 illustrates a schematic representation 
of the stereo camera system of Figure 1 having a second 
configuration of a stereo imaging means. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

10 Although this invention is applicable to 

numerous and various types of stereo imaging means for 
producing a stereo image, it has been found particularly 
useful in the environment of stereo camera systems having 
fixed mirrors or two or more cameras. Therefore, without 

15 limiting the applicability of the invention to stereo 
imaging means having fixed mirrors or two or more 
cameras, the invention will be described in such 
environment . 

Referring now to Figure 1, there is illustrated 
20 a schematic of a stereo camera system, generally referred 
to by reference numeral 100. The stereo camera system 
comprises a stereo imaging means 102 for outputting at 
least one stereo image of an object of interest 104 
located in the field of view of the stereo imaging means 
25 102. A recognition means locates the object of interest 
104 and calculates the distance of the object of interest 
104 from the stereo imaging means 102 and/or the size of 
the object of interest 104. An adjusting means 108 is 


US010059 


provided for automatically changing at least one system 
parameter which affects the spatial resolution of the 
object of interest 104 based on at least one of the 
located distance of the object of interest 104 from the 
5 stereo imaging means 102 and the size of the object of 
interest 104. Preferably, the adjustment means is under 
the control of a controller 110 which determines the 
amount or degree of adjustment of one or more of the 
system parameters based on the information from the 
10 recognition means 106. 

The recognition means 106 can detect the object 
of interest 104 according to prior knowledge of the 
object of interest 104 such as by its typical size, 
shape, and color. Such systems are well known in the art 

15 and include those described in, C. Wren et al., "Pfinder: 
Real-time Tracking of the Human Body/' IEEE Transaction 
on Pattern Analysis and Machine Intelligence (PAMI), 
19(7) :780-785, July 1997; H. Rowley et al., "Rotation 
Invariant Neural Network-Based Face Detection, " Proc. 

20 IEEE Conference on Computer Vision and Pattern 

Recognition, pp. 38-44, June 1998; and A. Lipton et al., 
"Moving Target Classification and Tracking from Real-Time 
Video," Proc. IEEE Workshop on Application of Computer 
Vision, pp. 8-14, Oct 1998. 

25 The recognition means 106 can also detect the 

object of interest 104 automatically, with the use of a 
stereo vision system which analyzes image data from the 
stereo imaging means 102. These types of recognition 
means 106 are preferred because the field of view can be 
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adjusted to cover a large three dimensional area and 
objects of interest 104 can be detected as foreground 
objects closest to the stereo imaging means 102. Stereo 
vision systems are well known in the art and generally 
5 operate by first recovering the internal parameters of 
the cameras, and external parameters between the cameras. 
For all stereo images taken at the same time, features 
are extracted and matched across different views. From 
the matching and the camera parameters, the depth of 
10 various points in the scene can be computed. A detail 
review of research work on stereo can be found in U.R. 

% Dhond et al., "Structure from Stereo - A Review", IEEE 

*y 

w Transaction on Systems, Man, and Cybernetics, vol. 19, 

Jf pp. 1489-1510, 1989. 

15 The stereo imaging means, adjustment means, and 

» controller will now be discussed in detail with regard to 

'?* a first and second configuration of the stereo camera 

system, illustrated in Figures 2 and 3, respectively. 

y- Referring now to Figure 2, there is illustrated 

20 a schematic of the stereo camera system 100 of Figure 1, 
and having a first configuration of the stereo imaging 
means 102. The stereo imaging means 102 in the first 
configuration illustrated in Figure 2 comprises a camera 
112 and a set of mirrors 114, 116. The mirrors 114, 116, 
25 are angled with respect to each other at a predetermined 
angle 9 and disposed a predetermined distance d from the 
camera 112. Such a stereo imaging means is well known in 
the art for producing a stereo effect in the output of 
the camera 112. The camera 112 can be a still camera 
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where the stereo image produced thereby is a still image 
or the camera 112 can be a video camera where the stereo 
image produced thereby is a sequence of video images* 

The adjusting means 108 in such a first 
configuration preferably comprises adjustment means 108a, 
108b, 108c for adjusting the predetermined angle 9=ai+a 2 
between the set of mirrors 114, 116; for adjusting the 
predetermined distance d between the camera 112 and the 
set of mirrors 114, 116; and for changing a focal length 
of the camera 112, respectively. As discussed above with 
regard to Figure 1, the controller 110 controls at least 
one of the angle, distance, and focal length adjustment 
means 108a, 108b, 108c, respectively, based on an input 
signal from the recognition means 106 containing 
information regarding how far the object of interest 104 
is from the camera 112 and/or the relative size of the 
object of interest 104. Preferably, a combination of all 
three adjustment means 108a, 108b, 108c are controlled to 
optimize the spatial resolution of the object of interest 
104. 

Given the configuration illustrated in Figure 
2, to increase the spatial resolution of a detected 
object of interest 104, the controller 110 would input 
the angle adjustment means 108a to decrease angle 0. 
Similarly, to increase the spatial resolution of a 
detected object of interest 104, the controller 110 would 
input the distance and focal length adjustment means 
108b, 108c, to decrease distance d and decrease the focal 
length of the camera 112, respectively. 
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The controller 110 can be any processor capable 
of performing the necessary calculations to determine the 
amount of adjustment to each of the adjustment means in 
order to increase and/or optimize the spatial resolution 
of the object of interest, such as a personal computer. 

Let B denote the baseline, which is the 
distance between two virtual cameras 114a, 116a, normal 
with a respective mirror 114, 116. 

B = 2Z c *$m(a l +a 2 ) ^ 

To increase resolution in depth, the disparity 
in range is maximized, denoted by DR, given the 
constraint that size of the image is fixed with width = 
Xres, height = Yres, and the operational range is 


DR = f-B-(— — ) (2) 

7 Z 


rain max 


Where f is the focal length of the camera 112, Z^± n and Z ma x 
are the minimum and maximum distance between the object 
of interest 104 and the camera 112. 


DR = f-Zc- sin(a 1 + a 2 ) ■ (y- - j-) 0) 

min max 

Assuming pinhole camera model: (and symmetrically for Y- 
coordinates) 
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mm 


(4) 


where dx = X -X 

max mm 

and X min and X max are the minimum and maximum values of x- 
coordinate of an object in the scene. 

5 

Putting equation (4) in equation (3) : 


Jres f , (5) 


5* 


DR = Zc- sin(a, +a 2 )-(-f-- —± ) 

Xres 

&i where ? 7 7 7 

fl aL ~ Z max _Z min 


10 Also, let cp denote the field of view of the camera 112, 
Then, 


And 


2 sin^ + a 2 - <p) cos(a 1 + a 2 ) sm(a 1 +a 2 -<p) 


15 


Therefore, according to equation (5) , to 
maximize the disparity range DR: 

(1) the distance Zc between the camera and the 
mirror can be increased, which would also increase the 
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minimum distance of the object 104 in scene from the 
camera 112; 

(2) the angle between the mirrors 0=ai+a 2/ can 
be increased which would also change the minimum distance 

5 of the object 104 in the scene from the camera 112; 
and/or 

(3) the focal length f of the camera can be 

decreased. 

Referring now to Figure 3, there is illustrated 

10 a schematic of the stereo camera system 100 of Figure 1, 
and having a second configuration of the stereo imaging 
means 102. The stereo imaging means 102 in the second 
configuration illustrated in Figure 3 comprises first and 
second cameras 120, 122. The first and second cameras 

15 120, 122 are angled a predetermined angle |3 with respect 
to each other. Each of the first and second cameras 120, 
122 are disposed a predetermined distance dl, 62, 
respectively, from the object of interest 104 and the 
cameras are spaced apart a predetermined distance B 

20 (generally referred to as the baseline distance) . Two 

cameras are illustrated by way of example only and not to 
limit the scope or spirit of the present invention. 
Those skilled in the art will realize that more than two 
cameras can be utilized to produce a stereoscopic image. 

25 Furthermore, as discussed above with regard to the first 
configuration illustrated in Figure 2, each of the 
cameras 120, 122 can be either a still image or video 
image camera. 
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The adjusting means 108 in such a second 
configuration preferably comprises adjustment means 108d, 
108e, 108f, 108g for adjusting angle a by adjusting the 
angle of at least one of the first and second cameras 
5 120, 122, for adjusting the baseline distance b between 
the first and second cameras 120, 122, for adjusting the 
distance dl, d2 between either or both of the first and 
second cameras 120, 122 and the object of interest, and 
for changing a focal length of at least one of the first 
10 and second cameras 120, 122, respectively. 

As discussed above with regard to Figure 2, the 
controller 110 controls at least one of the angle, 
baseline, distance, and focal length adjustment means 
108d, 108e, 108f, 108g, respectively, based on an input 
signal from the recognition means 106 containing 
information regarding how far the object of interest 104 
is from the camera 112 and/or the relative size of the 
object of interest 104. Preferably, a combination of all 
four adjustment means 108d, 108e, 108f, 108g are 
controlled to optimize the spatial resolution of the 
object of interest 104. 

Given the configuration illustrated in Figure 
3, to increase the spatial resolution of a detected 
object of interest 104, the controller 110 would input 
25 the angle adjustment means 108d to increase angle a. 
Similarly, to increase the spatial resolution of a 
detected object of interest 104, the controller 110 would 
input the baseline, distance, and focal length adjustment 
means 108e, 108f, 108g to decrease the baseline distance 
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B, to decrease distances dl and/or d2, and to decrease 
the focal length of the camera 112, respectively • 

As discussed above, the controller 110 can be 
any processor capable of performing the necessary 
5 calculations to determine the amount of adjustment to 

each of the adjustment means in order to increase and/or 
optimize the spatial resolution of the object of 
interest, such as a personal computer. 

In the case where two or more cameras 120, 122 
10 are used, equations (2) and (4) above can be used to 

determine the amount of adjustment necessary to increase 
disparity range DR according to equation (2) . For 
instance, the distance B between the cameras 120, 122 can 
be increased and or the focal length f of the cameras 
15 120, 122 can be increased, which would increase the 

minimum distance of the object 104 in the scene from the 
cameras 120, 122. 

The adjustment means 108 for adjusting the 
system parameters such as angles 9 and (3 and distances d, 

20 dl, d2, and B are well known in the art and a detailed 

description is therefore omitted for the sake of brevity. 
Those skilled in the art realize that such adjustment 
means can be accomplished by way of linear and rotary 
motion devices such as linear screws and belt drives and 

25 rotary stepper or servo motors, respectively, which are 
appropriately interconnected with the intended structure 
to be adjusted. 
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Those skilled in the art will appreciate that 
the stereo camera system 100 of the present invention 
automatically determines the necessary operational range 
for the object of interest, and adjusts stereo system 
parameters to achieve better three dimensional spatial 
resolution. Such an adjustable stereo camera system 100 
can provide improved resolution as a preprocessing step 
for further image analysis steps which demand good 
resolution in three dimensions (x, y, and depth) , such as 
face, gesture, and body recognition using three 
dimensional inputs . 

Furthermore, the adjustable stereo camera 
system 100 of the present invention can isolate a volume 
of interest from the rest of a scene, for example, a 
person in front of the system 100 and interacting with it 
in a public environment. The stereo camera system 100 
can provide better spatial resolution for analysis of the 
person and his or her immediate surroundings while 
eliminating any background activity which can distract or 
corrupt the analysis. 

These objectives and advantages of the present 
invention can be achieved by a stand-alone system such as 
those illustrated in the Figures or a system which adapts 
to and interfaces with exiting stereo imaging means such 
as those described in Figures 2 and 3. 

While there has been shown and described what 
is considered to be preferred embodiments of the 
invention, it will, of course, be understood that various 
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modifications and changes in form or detail could readily 
be made without departing from the spirit of the 
invention. It is therefore intended that the invention 
be not limited to the exact forms described and 
illustrated, but should be constructed to cover all 
modifications that may fall within the scope of the 
appended claims. 
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